DOCUMENT RESUME 



ED 392 A21 



IR 017 712 



AUTHOR 

TITLE 

PUB DATE 
NOTE 

PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Kay, Robin H* 

Identifying Effective Knowledge Building Activities 
for Learning Computer Software* 

95 

9p*; In: **Emerging Technologies, Lifelong Learning, 
NECC '95**; see IR 017 705* 

Reports Research/Technical (1A3) — 
Speeches/Conference Papers (150) 

MFOl/PCOl Plus Postage* 

Adult Education; Cognitive Style; ^Computer Software; 
*Epistemology; Foreign Countries ; ^Learning 
Processes; Protocol Analysis; Spreadsheets 
Computer Use; ^Knowledge Acquisition 



ABSTRACT 

This paper details a study of knowledge-building 
activities in the domain of computer software* Thirty-six 
adults — evenly split among beginning, intermediate, and advanced 
knowledge levels — were videotaped while they attempted to learn a 
common spreadsheet package* Subjects were asked to think aloud while 
learning* Each knowledge building activity was given a score based on 
how much was learned and how much time was gained or lost* 

Tabulations revealed that directed search, trial and error, and 
careful observation had the strongest impact on learning; pace of 
learning and systematic testing had a more moderate effect* From 
these results, a list is drawn of suggestions for educators who are 
trying to learn software, including making searches for help more 
focused instead of wasting time with frantic random keystrokes* Three 
tables summarize the data* (Contains 18 references*) (BEW) 



Vc Vc Vc Vf Vc Vf Vc Vc Vc Vf Vc Vc Vc Vc Vc Vc "ft V? Vf Vc Vc Vc Vf Vc Vc Vc ic Vc it it it it it it it it it it it it it it it it it it it it it it it it it it it it it itit^tif>t^tit^t^t^t^t^t^t^t 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document* 

* it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 



o 

ERIC 






A 



<N 

fS 

ON 

cn 

a 

pj 



CENTER (ERIC) 

° :eTol“"epS'o?;?9n^^^^^ 
originating it 

a Minor changes have been made to 
i.nprovc reproduction quality 

• Points of view or 

document do not necessan y reprcscni 
official OERI position or policy 



Identifying Effective Knowledge Building 
Activities for Learning Computer 

Software 

by Robin H. Kay 



Paper presented at the NECC ‘95, the Annual National Educational 
Computing Conference (16th, Baltimore, MD, June 17-19, 1995. 



■■PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Done 11a Ingham 




BEST COPY AVAIUBLE 



o 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



paper 

Identifying Effective Knowledge Building Activities for Learning 
Computer Software 

Robin H. Kay 

118 Gloucester Avenue 

Oakville, Ontario, M8X 1A3 

CANADA 

(416) 844-6970 

RKAY@OISE.ON.CA 

Key words: learning, software, computers, strategies, knowledge-building, teacher- 
education 

Abstract 

This Study looked at knowledge building activities (KBAs) in the domain of computer software. Thirty-six adults were 
asked to think aloud while learning a series of tasks in a common spreadsheet package. A detailed analysis of 3,061 KBAs 
revealed that directed search, trial and error, and careful observation had a strong impact on learning. Pace of learning and 
systematic testing played a more moderate role. Practical recommendations are offered to educators of computer studies. 

Introduction 

The exploration of knowledge building activities (KBAs) originates from task-analysis in human factors engineering 
(Drury, Paramore, Van Cott, Grey, & Corlett, 1987) and involves breaking down tasks into smaller components and specifying 
the flow of those components (Brooks, 1991). A partial list of KBAs that have been identified in a variety of domains includes 
defining terms (Bransford, Vye, Adams, & Perfetto, 1989), planning (Miyata & Norman, 1986, Riley, 1986; Voss, 1989), search- 
ing (Schauble & Glaser, 1990), questioning (O'Malley, 1986), and a mixture of successful strategies (rehearsal and review, 
monitoring, unpacking implicit assumptions, summarizing, considering alternatives, questioning, clarifying and predicting) 
(Bereiter & Scardamalia, 1989; Brown & Palinscar, 1989; Collins, Brown, & Newman, 1989). 

A number of investigators in the area of education and cognitive science have argued for more research on the knowledge 
building process. Chi & Bassok (1989) maintain that there is a need for more explicit examples describing the conditions 
surrounding KBAs. Glaser (1990) and Glaser & Bassok (1989) note that while considerable advances have been made in the 
areas of memory organization, problem solving, and characteristics of understanding, the knowledge acquisition process has 
not been examined extensively. Furthermore, few researchers have offered a detailed analysis of knowledge building charac- 
teristics in a natural setting (Ceci, 1990; Siegler, 1989). 

Research on KBAs has been noticeably absent in the computer ability literature. For the most part, studies have relied 
solely on a paper-and-pencil format to gather information about knowledge of using computers. This relatively straightfor- 
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ward strategy has produced a wealth of well-organized, albeit conflicting, information (Kay, 1989, in press) offering little 
understan-Hing of the dynamics of human-computer interaction. 

The primary purpose of this study was to examine knowledge building activities used to learn new computer software. 
The specitr* objectives were to (a) describe the principle KBAs in the computer knowledge acquisition process, (b) examine the 
extent to 'vhich these activities were associated with successful problem solving; (c) demonstrate how the msults can be used 
to adv ance knowledge and theory; and (d) prov ide meaningful suggestions for educators of computer studies. 



Research Design 

Sample 

The sample consisted of 36 adult volunteers (18 male, 18 female): 12 beginners, 12 intermediates, and 12 advanced users, 
ranging in age from 23 to 49 (M= 33.0 years), living in the greater metropolitan Toronto area. The criteria used to determined 
ability levels included years experience, previous collaboration, learning, software experience, number of application software 
packages used, number of programming languages/operating systems known, and application software and programming 
languages known. A MANOVA showed that beginners, intermediates, and advanced users had significantly different scores 
in the expected direction on all seven criteria (p <.005). 

Procedure 

Subjects' computer activities were videotaped for 55 minutes, with the camera focused on the screen, while they at- 
tempted to learn a common spreadsheet package (Lotus 1-2-3, Version 2.2) on an IBM 80286 clone. Subjects were asked to do 
as many of the following tasks as they could: (1) move the cursor, (2) enter rows and columns of numbers, (3) enter data, (4) 
insert blank rows and columns, and (5) move and/or copy rows or columns of data. The standard procedure was to introduce 
the subject to a task and encourage self-directed learning. Each subject was asked to think out loud while learning. Every effort 
was made to encourage subjects to get "unstuck" on their own. Unlike typical protocol analysis, subjects were given calculated 
"hints" when they w'ere unable to proceed. 

Data Collection 

The first 30 minutes of each of the 36 videotaped sessions was transcribed verbatim. Verbal expressions and sounds, as 
well as critical kevstrokes, were included in the transcriptions. Transcription analysis revealed that 3,061 learning episodes 
invoK ed some sort of knowledge building activitv'. 

Each KBA was assigned an influence rating based on (a) how much was learned and (b) time lost or gained. If no knowl- 
edge or understanding was lost or gained, a score of 0 was given. If a small piece of the task was solved or misconstrued a 
score of -^l or -1 was assessed respectively. If a significant piece of the task was learned or misconstrued, a score of +2 or -2 was 
assessed. Finally if a substantial amount of time (15-20 minutes) was gained or lost, in the process of learning or misconstru- 
ing, a score of +3 or -3 u'as given. A detailed coding scheme was used to ensum the reliability t>f influence ratings. 

Next, an estimation of effect for a specific category of KBA was calculated using the average influence rating for that 
category, the percentage of subjects im'oK'ed, and the total number of observations made. For example, if the KBA category 
"seeking information" showed an average influence rating of 1.2, was used by 50‘’o of the subjects (n=l8), and accounted for a 
total of 45 observations, the estimated effect scorn would be 27 (1.2 x .50 x 45). 

Main Variables 

The main independent variables were based on five principle KBAs identified in the protocol data: (a) actions (e.g. 
pressing a key, playing, exploring), (b) seeking information, (c) processing information, (d) st\ le (e.g. going at a fast or slow 
pace), and (e) combinations of learning acti\'ities. Within these categories, a number of sub-categories u ere identified and are 
presented in Table 1. The main response variable vvas the estimated effect score. 



Results 

Description of Principle KBAs 

Seeking information (48"«) and specific actions (36"o) were the two most prolific KBA categories. Main acti\ ities based on 
style (9‘*«), processing of knowledge (5“o)» and combinations of activities (1“(» ) were seen much less often . Trial and error, 
searching (specific and broad), and observing were the top sub-categories of learning observed. Subjects' pressed any key, 
repeated their actions, performed svstematic tests, asked for help, used deduction and changed their pace of learning with a 
moderate degree of frequency. 
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Table 1: Type and Frequency of Knowledge Building Activities 



CATEGORY 


# of Obs, 


% of Category CATEGORY 


# of Obs, 


% of Catego 


ACTIONS 






KNOWLEEX3E 












PROCESSING 






Thai & Error 


608 


54^> 


Deduce 


106 


64% 


Pre^ii Any Key 


127 


ir’o 


Reflect 


42 


25% 


Repeat Action 


114 


10% 


Compare 


18 


U% 


Si/s. Test 


114 


10"« 


TOTAL 


166 


5%** 


Exploring 


63 


6“o 








At ichor in g 


62 


5"o 


STYLE 






Playing 


25 


2% 


Fast-Pace 


118 


4\% 


Circling 


15 


1% 


Slow-Pace 


101 


35% 


TOTAL 


1128 


36%** 


Double-Check 


61 


21% 








Evaluate 


11 


4% 


SEEK INFORMATION 




TOTAL 


291 




Search Specific 


566 


38% 








Search Broad 


454 


31% 


Combining KBAs 


32 


1%** 


Observe 


351 


24% 








Ask for Help 


108 


7% 








TOTAL 


1479 


48%** 









Note: Percent of all KB As 



Estimated Effect of KBAs on Learning 

The highest effect score came from attempts to seek information (990.9) and actions subjects made (699.4 \ Searching for 
specific information produced the single highest effect score (600.0) for a specific learning activity, followed by trial and error 
(486.4) and observing (358.0). More modest effect sizes were seen in broad searches for information (140.7), a slow and fast 
pace of learning (100.3 and -115.1 respectively), and systematic testing (99.6). Notable negat^'e effects were produced by 
pressing any key (-55.8) and going at fast pace (-115.1) 

It is interesting to note the highest mean influence scores were not necessarily produced i y those activities with the 
highest effect scores. In fact, combining KBAs showed the highest mean influence (M=1.56) followed by change of pace 
(M=1.27 & 1.25), double-checking actions (M= L21), and systematic testing (M=1.21). Two of tht highest estimated effect 
activities, specific searches and observing had amatively high mean influences (M=1.06 and 1.02 respectively). Other relatively 
high influence activities included comparing (M=1.00) and asking for help (M=1.06). A complete set of estimated effects for 
main and sub-categories of learning activities is presented in Table 2. 
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Table 2: Estimated Effect as a Function of KBAs 



CATEGORY 


n 


% of Subjects 


Mean Influence 


Estimated Effect 


ACTIONS 










Trial & Error 


60? 


100% 


0.80 


486.4 


Systematic Testing 


114 


78% 


1.12 


99.6 


Repeat Action 


114 


92"/.. 


0.53 


55.6 


Exp>loring 


63 


86% 


0.79 


42.8 


Anchoring 


62 


69"/.. 


0.74 


31.7 


Playing 


25 


33"/., 


0.36 


3.0 


Circling 


15 


28"'., 


-0.47 


-2.0 


Press Any Key 


127 


72"'.. 


-0.61 


-55.8 


TOTAL ' 


1128 


100"/o 


0.62 


699.4 


SEEK INFORMATION 










Search for Specific Info. 


566 


100"/o 


1.06 


600.0 


Observing 


351 


100"/., 


1.02 


358.0 


Search Broad 


454 


100"/. 


0.31 


140.7 


Ask for Help 


108 


78"/., 


1.06 


89.3 


TOTAL 


1479 


100"/, 


0.67 


990.9 


KNOWLEDGE PROCESSING 








Deduce 


106 


92"/., 


0.72 


70.2 


Reflect 


42 


69"/., 


0.81 


23.5 


Compme 


18 


36"/., 


1.00 


6.5 


TOTAL 


166 


100"/o 


0.77 


127.8 


STYLE 










Slow-Pace 


101 


86% 


1.27 


110.3 


Double-Check 


61 


78"/o 


1.21 


57.6 


Evaluate 


11 


177o 


0 


0.0 


Fast-Pace 


118 


78"'o 


-1.25 


-115.1 


TOTAL 


291 


97% 


0.19 


53.8 


COMBINING KBAs 


32 


33% 


1.56 


16.5 


ALL KBAs 


3061 


1007. 


0.60 


1836.6 



Finally, an analysis of KBAs as a function of ability showed advanced users to be more capable with respect to trial and 
error, observation, searching for information, and double checking their answers. Paradoxically, they experienced more 
difficulty than beginners or intermediates when they attempted to go too fast. Estimated effects for all learning activities as a 
function of cbility are present in Table 3. 
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Table 3: Estimated Effect as a Function of Learning Activity and Ability Level 



Learning Activities 


Beginner 


Intennediate 


Advanced 


(from Resources) 
ACTIONS 


Est. Effect 


Est. Effect 


Est. Effect 


Amliorin};* 


7.4 


14.2 


10.7 


Circlini;^ 


-1.0 


-0.5 


-0.5 


Explorui^ 


18.3 


15.6 


9.7 


Press any key 


-23.9 


-17.3 


-14.2 


Repeat an action 


15.6 


14.2 


25.9 


Systematic test 


30.7 


45.0 


24.7 


Trial & Error 

SEEKING INFORMATION 


146.3 


143.5 


197.8 


Ask for help 


29.3 


26.5 


32.5 


Observe 


110,7 


108.9 


138.8 


Search Specific 


71.8 


128.5 


194.9 


Search Broad 

KNOWLEDGE PROCESSING 


20.6 


34.3 


42.3 


Coniparin<^ 


0.3 


3.0 


4.2 


Deducinif 


24.9 


30.0 


15.8 


Reflecting 

STYLE-RELATED 


8.6 


5.5 


9.2 


Double-check 


10.0 


16.4 


33.8 


Ezuiluate software 


-0.1 


0.0 


0.5 


Fast-Pace 


-16.2 


-44.9 


-54.0 


Slow-pace 


33.7 


37.5 


38.0 


COMBINATION 


0.5 


1.5 


21.0 



Theoretical Implications 

Identification of KB As for computer knowledge acqiiisition has proven to be a useful procedure yielding a mixture of 
traditional and unique behaviors. KBAs observed in this study that wem noted previously in other domains included search- 
ing (Schauble & Glaser, 1990), monitoring or observing, and review (double checking) (Bereiter & Scardamalia, 1989; Brown & 
Palinscar, 1989; Collins, Brown, & Newman, 1989). Iraditional KBAs, such as rehearsal, unpacking of implicit assumptions, 
summarizing, considering alternatives, clarifying, and pmdicting (Bereiter & Scardamalia, 1989; Brown & Palinscar, 1989; 
Collins, Brown, & Newman, 1989) were not seen frequently in this investigation. Some knowledge processing activities and 
strategies, thmigh, such as deduction, reflecting, and comparing were observed occasionally, with modest effects on learning. 

Influential KBAs, seemingly indigenous to the computer environment, included trial atui error and sifsteniatie L.ess 

powerful KBAs, also unique to softvvare use wore anchoring (always comiiig back to the same spot) and random, rapid /mNsi/jy 
ofanu keif. In addition, pace of KBAs, categf ’ ized under style, appeared to be important in learning. Fa^t key pre^^in^, quirk 
searehin^ or liU'^ty observation^ lead to mistakes and a negative effects on learning. Slowing down on the other hand increased 
lea rni ng effectiveness. 

The nature of the domain iii this study may have partially determined the nature of KBAs expressed. Computer software 
is set up to naturally try keys, and offer instant feedback, usually with minimal repercussions. A trial and error str itegy can bi* 
useful tor providing new information and constraii/s to guide future actions and learning. While the mean influence of trial 
and error activity is oiily moderately high, instant feedback makes it a useful strategy, when employed frequently and with a 
certain amount of prudence. Aimless key pressing or rushing on to the next key has a definite negative efh*ct on learning, 
primarily because the subject caii neither see nor cognitively digest what has happened. Systematic testing and taking the 
necessary time to observe and interpret feedback, is an optimal approach. E^xhaustive searches of the manual or on-line help, 
or careful planning and deduction, may not appear worthwhile given the relative efficiency of trying specific keys in a • 
constrained and systematic manner. I he potential for considerable time li^ssand absence of immediate feedback, makes the 
trial and error strategy somewhat inefficient for non-computer software domains. 

rile quality of searching seems to be fundamental to success in learning with computers. Specific searches were more than 
four times more effective (600 to 141) and three times more efficient (1.06 to 0.31) than broad searches. Specific searching is 
characterized by searching for meaning or specific phrases, whereas broad searching involved general scanning and page 
turning. It is n*asonable to assume that a more specific search strategy is linked to ability level and more specifically to one's 
understanding of terminology and concepts. Advanced users have a better iilea of what to look for, whereas a beginners, are 
sometimes torced to search broadly, because\hey do not know where to focus their efforts. 

riie key KBAs noted above were instrumcHital in distinguishing advanced us(.’r from their less able counterparts: trial and 
error, observation, searching for information, and double checking their answers. Somewhat unexpectedly, advance users have 



Fa^e 6*8 



National l.ilueational Computing Conferenee, /993 



an Achilles heel-they attempt to go too quickly and their learning suffers as a result. 

One final observation: although the number of times subjects' combined strategies was minimal (n=32), the mean influ- 
ence of this KB A was the highest observed. Ultimate success with learning new computer software may rest on combining the 
strategy of slowing down, with careful observation, searching as specifically as possible, and trying keys in a systematic 
manner. 



Suggestions for Educators 

The findings in this section suggest several clear guidelines for educators of computer studies. These include: 

• A trial and error strategy appears to work well.' 

• The effectiveness of trial and error is increased by using a more systematic approach. Pressing any key randomly is 
unlikely to help learning and will moderately impede progress. 

• Taking time to observe not only key strokes, but what is on the screen and in the book appears to be highly related to 

success. 

• When searching a manual, on-line help, or a menu, specific searches are four times more effective and a three times 
more efficient than broad searches (e.g. turning pages, scanning). ActiviHes supporHng the labeling of new acHons and 
concepts might support a more specific search technique. 

• Deduction, reflection, and comparing do not appear to play a important role in short term learning of new software. 

• A combination of the above strategies may be the most effective way to improve learning with computers. In other 
words, each additional piece may increase a student's overall chances of success. 



Caveats 

Several cautions should be noted with respect to the results of this study. First, the rating system focused on short-term 
leaming-the conclusions do not necessarily apply to long term gains. Second, only one software package was assessed- 
different software areas might require different KBAs. Finally, w'hile over 3000 observations were made, the sample size (n=36) 
w'as relatively small and localized. 

Summary 

Knowledge building activities play a prominent role in computers and learning. While certain KBAs observed, such as 
searching, monitoring, and review were observed previously in other domains, KBAs such as trial and error and systematic 
testing appeared to be unique to the computer environment. KBAs noted in other subject areas, such as rehearsal, planning, 
unpacking assumptions, and summarizing played a negligible role in this study. It was suggested that the nature of the 
computer software domain may have effected the frequency and quality of KBAs observed. An examination of all learning 
activities indicated that a slow, deliberate, trial and error strategy, coupled with observation, and more focused searching 
techniques would be an optimal approach to learning new software. 
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